node 2 still down, now avoiding - crashed again

Gavery NGS running on node 4

NCBI all Crassostrea running on node 3

------

8am next day

Node 4- not running

Node 3- still running

----

6am day 2 (092910)

node 3 still running

node 4 ended at  10797

ERROR for node 4

bigfish:~ safs$ mpiexec -np 2 -host localhost,node004 /common/clcbfxcell/blastall_cell_mpi -c cell_sw -v 1 -a 16 -b 1 -p blastx -m 8 -i /Volumes/bigfishRAID/Users/safs/Dropbox/Cluster/tmp/GaveryNGS163263contigs.fa -d /common/clcbfxcell/databases/swissprot.fasta > /Volumes/bigfishRAID/Users/safs/Dropbox/Cluster/tmp/GaveryNGS163263contigs_swiss.txt

[node004:48454] *** Process received signal ***

[node004:48454] Signal: Segmentation fault (11)

[node004:48454] Signal code: Address not mapped (1)

[node004:48454] Failing at address: 0x0

[node004:48454] [ 0] 2   libSystem.B.dylib                   0x0000000083c663fa _sigtramp + 26

[node004:48454] [ 1] 3   blastall_cell_mpi                   0x00000000006ab5e8 _ZTS24CFixClockSecureStoreFile + 567019

[node004:48454] [ 2] 4   blastall_cell_mpi                   0x0000000000316313 _mh_execute_header + 3236627

[node004:48454] [ 3] 5   blastall_cell_mpi                   0x000000000030d1ee _mh_execute_header + 3199470

[node004:48454] [ 4] 6   blastall_cell_mpi                   0x000000000000ccad _mh_execute_header + 52397

[node004:48454] [ 5] 7   blastall_cell_mpi                   0x000000000000e7f1 _mh_execute_header + 59377

[node004:48454] [ 6] 8   blastall_cell_mpi                   0x000000000000ed0a _mh_execute_header + 60682

[node004:48454] [ 7] 9   blastall_cell_mpi                   0x0000000000006c3b _mh_execute_header + 27707

[node004:48454] [ 8] 10  blastall_cell_mpi                   0x0000000000008094 _mh_execute_header + 32916

[node004:48454] [ 9] 11  blastall_cell_mpi                   0x00000000002918d2 _mh_execute_header + 2693330

[node004:48454] [10] 12  blastall_cell_mpi                   0x0000000000001864 _mh_execute_header + 6244

[node004:48454] [11] 13  ???                                 0x0000000000000011 0x0 + 17

[node004:48454] *** End of error message ***

--------------------------------------------------------------------------

mpiexec noticed that process rank 1 with PID 48454 on node node004 exited on signal 11 (Segmentation fault).

--------------------------------------------------------------------------

bigfish:~ safs$ 

____

restarted Gavery job on node 2